gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
5619 commits 28 branches 278 tags 135 MiB
Commit graph

252 commits

Author SHA1 Message Date
Chris Lu ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu 48ed7f8a32 generated files 2020-08-16 21:25:24 -07:00
Chris Lu f5d6b8b777 Hadoop HCFS 1.4.6 random access file 2020-08-16 21:17:03 -07:00
Chris Lu 13bfe5deef same logic for reading random access files from Go 2020-08-16 21:14:39 -07:00
Chris Lu 20e0bae5d1 add for testing 2020-08-16 09:03:18 -07:00
Chris Lu 24bfd26719 make it easy to test randome access file 2020-08-15 14:37:07 -07:00
Chris Lu 4f195a54ca 1.4.5 
change the scope for hadoop depencies to provided
 2020-08-09 16:34:47 -07:00
Chris Lu 15dc0a704d Revert "add read ahead input stream" 
This reverts commit b3089dcc8e.
 2020-08-03 09:11:24 -07:00
Chris Lu b3089dcc8e add read ahead input stream 2020-08-03 09:06:09 -07:00
Chris Lu 53190a9972 fix compilation 2020-08-03 09:05:49 -07:00
Chris Lu d6073f6386 ensure GC 2020-08-03 00:40:23 -07:00
Chris Lu 9981748498 only try to cache chunk manifest chunks 2020-08-02 23:50:44 -07:00
Chris Lu 0ed1f43d29 decompress after decrypt if necessary 
skip any decompress error
 2020-08-02 23:50:23 -07:00
Chris Lu e1f070a9a1 Hadoop: 1.4.4 2020-08-01 12:42:41 -07:00
Chris Lu 4d9da157bc HDFS: read gzip content 2020-08-01 11:36:29 -07:00
Chris Lu 8cc35e2c13 fix wrong decompression logic 2020-08-01 11:19:57 -07:00
Chris Lu 6b41c5250b Hadoop file system: 1.4.3 
added buffered fs input stream
 2020-07-29 21:33:10 -07:00
Chris Lu 703057bff9 mirror changes from hdfs2 2020-07-29 20:15:43 -07:00
Chris Lu a6d0f962e8 add buffered seaweed input stream 2020-07-29 20:10:04 -07:00
Chris Lu 5080bc1d69 debug 2020-07-29 18:26:26 -07:00
Chris Lu daf8c0c8ce cache vid locations 2020-07-29 18:25:52 -07:00
Chris Lu b684c312d2 minor 2020-07-29 18:25:35 -07:00
Chris Lu 735912e0f5 enable read cache 2020-07-29 17:39:44 -07:00
Chris Lu 6f058b3791 HCFS: 1.4.1 2020-07-22 22:56:53 -07:00
Chris Lu 648ef566c4 HCFS: avoid lock bottleneck 2020-07-22 22:54:06 -07:00
Chris Lu 6839f96c0c simplify 2020-07-22 22:52:49 -07:00
Chris Lu 54f9f9138e HCFS: 1.4.0 2020-07-20 23:25:08 -07:00
Chris Lu 64926d2345 disable ChunkCache 2020-07-20 21:18:58 -07:00
Chris Lu ae3e6d8244 remove changing buffer size 2020-07-20 20:27:19 -07:00
Chris Lu b380b9e5d7 shared http client 2020-07-20 18:27:11 -07:00
Chris Lu 0e341a2a9a error logging 2020-07-20 18:26:59 -07:00
Chris Lu cacc601cc8 ensure changing buffer size requirements 2020-07-20 18:26:48 -07:00
Chris Lu aee6d89350 before writing and after reading file chunks 2020-07-20 18:26:17 -07:00
Chris Lu 6debe3c3ec use 1000 as merge factor 2020-07-20 18:25:26 -07:00
Chris Lu 1d724ab237 hdfs: support read write chunk manifest 2020-07-20 03:29:17 -07:00
Chris Lu f90d2c93c9 1.3.9 remove logs 2020-07-17 16:11:57 -07:00
Chris Lu 3abd74b1d7 1.3.8 2020-07-17 15:46:20 -07:00
Chris Lu 0b9d73fb7c async copy 2020-07-17 15:46:20 -07:00
Chris Lu 54445b207d Hadoop: 1.3.6 2020-07-16 16:20:01 -07:00
Chris Lu d6d95765f9 reduce chunk cache size 2020-07-16 16:19:01 -07:00
Chris Lu 2629da2cb9 simplify inputstream 2020-07-16 10:54:29 -07:00
Chris Lu c9b50e8a22 copy changes to hadoop2 2020-07-16 09:05:34 -07:00
李明达 74456b3d5e 1、Add SeaweedFS implementation of Hadoop AbstractFileSystem. The implementation delegates to the existing SeaweedFS FileSystem and is only necessary for use with Hadoop 2.x/3.x. Configuration example in Hadoop core-site.xml file: 
<property>
    <name>fs.AbstractFileSystem.seaweedfs.impl</name>
    <value>seaweed.hdfs.SeaweedAbstractFileSystem</value>
 </property>

2、Fix hiveserver2 startup NullPointerException
 2020-07-16 16:47:09 +08:00
Chris Lu 64d7d5fe89 increase inbound message size 2020-07-16 00:33:54 -07:00
Chris Lu 7bca72deed reuse bytebuffer 2020-07-15 23:33:31 -07:00
Chris Lu 2286d27730 rename 2020-07-15 14:19:15 -07:00
Chris Lu 86c8137546 1.3.4 2020-07-15 13:49:06 -07:00
Chris Lu 22f8aae028 fix compilation 2020-07-15 13:27:10 -07:00
Chris Lu bc3be0bb37 Hadoop: 1.3.3 
improve memory efficiency
 2020-07-15 13:25:44 -07:00
Chris Lu 316e853e0e option to disable chunk cache 2020-07-15 09:51:51 -07:00
Chris Lu 0db4204c81 add is_file_chunks to filer.proto 2020-07-14 22:42:52 -07:00
Chris Lu fc9af962ec 1.3.2 2020-07-12 14:51:24 -07:00
Chris Lu 4bd8f3281e 1.3.0 
* add option to include last entry
*
 2020-07-11 22:24:02 -07:00
Chris Lu 578f316173 refactoring 2020-07-11 15:33:31 -07:00
Chris Lu 70d8a3a1d3 add SubscribeLocalMetadata without checking persisted meta logs 2020-07-05 15:50:07 -07:00
Chris Lu 37d5b3ba12 replication: pass isFromOtherCluster also to EventNotification 
EventNotification is consistent with message queue and metadata logs.
 2020-07-01 08:06:20 -07:00
Chris Lu 31e23e9783 filer: support active<=>active filer replication 2020-06-30 22:53:57 -07:00
Chris Lu 97239ce6f1 rename filechunk is_gzipped to is_compressed 2020-06-20 08:15:49 -07:00
Chris Lu ca3516ac6d adjust protoc 2020-06-20 08:00:25 -07:00
Chris Lu ce79ec0bd4 HCFS: 1.2.9 fix concurrent modification 2020-06-18 07:52:32 -07:00
Chris Lu 222f93e816 possibly fix concurrent access to entry object in SeaweedOutputStream 2020-06-17 14:59:43 -07:00
Chris Lu 8dfaaeabfd HCFS: 1.2.8 fix hbase related bugs 
1. SeaweedFileSystem.listStatus need to work with file also
2. SeaweedRead readChunkView has wrong len
 2020-05-24 17:07:34 -07:00
Chris Lu dfccc3c263 able to read chan and write chan 2020-05-08 02:47:22 -07:00
Chris Lu 1e3e4b3072 add broker connects to filer 2020-05-05 02:05:28 -07:00
Chris Lu 972f437225 HCFS 1.2.7 2020-04-26 05:44:56 -07:00
Chris Lu b52b8ec685 Hadoop: fix entry not found for HCFS 
also fix cipher related changes.
 2020-04-26 05:21:54 -07:00
Chris Lu e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu bda82f61bc filer: able to append to a file 2020-04-17 02:28:09 -07:00
Chris Lu 7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu 6f948e4887 remove configurable topics folder location 2020-04-12 13:07:59 -07:00
Chris Lu ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu 2d43f85577 watch entries with common path prefix 2020-04-05 12:51:21 -07:00
Chris Lu bf270d9e8c filer: able to tail meta data changes 2020-04-05 00:51:16 -07:00
Chris Lu 50a5018b7f writing meta logs is working 2020-03-30 01:19:33 -07:00
Chris Lu 5ebf232e79 HDFS: 1.2.6 2020-03-29 01:40:04 -07:00
Chris Lu b656e05aaf HDFS: support chunk cache 2020-03-29 00:55:40 -07:00
Chris Lu ae2309dc58 Update SeaweedRead.java 2020-03-28 23:58:59 -07:00
Chris Lu 3d3bab2447 add notes 2020-03-28 23:43:24 -07:00
Chris Lu c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu de1ba85346 HDFS support encrypted data storage 2020-03-14 00:27:57 -07:00
Chris Lu 2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu 13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu 555413d9fc weed queue starts 2020-02-27 00:07:13 -08:00
Chris Lu 0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu 6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu d8dec2323b s3: move buckets folder configuration to filer 2020-02-24 14:34:14 -08:00
Chris Lu c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu 6a5c037099 fix http range requests 2020-01-22 22:59:12 -08:00
Chris Lu d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu 69ffcbdcc0 HCFS: release 1.2.4 2019-12-14 11:04:36 -08:00
Chris Lu 987108a2b1 HCFS: streaming list files 2019-12-13 10:04:53 -08:00
Chris Lu 0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu e9dd5a602a HCFS: 1.2.3 
shade org.apache.http in https://github.com/chrislusf/seaweedfs/issues/1080
 2019-10-11 23:56:04 -07:00
Chris Lu 7057e7a076 HCFS: 1.22 
fix put gives java.lang.NoSuchFieldError: INSTANCE related to Cloudera CDH Hadoop https://github.com/chrislusf/seaweedfs/issues/1080
 2019-10-11 21:44:44 -07:00
Chris Lu 15f968b054 HCFS: avoid possible jar problem 2019-10-09 18:09:08 -07:00
Chris Lu a999ed94d0 update hdfs client 2019-10-08 22:49:10 -07:00
Chris Lu ae53f63680 filer: recursive deletion optionally ignoring any errors 
fix https://github.com/chrislusf/seaweedfs/issues/1062
 2019-09-11 20:26:20 -07:00
Chris Lu d361ecb23b HCFS: 1.1.6 2019-09-03 01:31:31 -07:00
Chris Lu fd509c3844 HCFS: working with HBase 2019-09-03 00:50:28 -07:00
Chris Lu 60c9215a00 HCFS: avoid verbose file not found exception stack trace 2019-09-01 21:40:26 -07:00
Chris Lu af8b413a9c HCFS: fix mkdirs NPE 2019-09-01 12:25:06 -07:00
Chris Lu cb299dfaa2 HCFS: use latest grpc versions, separate hadoop2 and hadoop3 2019-09-01 03:46:51 -07:00
Chris Lu 170ee6ef0f tmp 2019-08-29 23:29:10 -07:00
Chris Lu 8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu 82b0759493 filer: migrating filer store from persisting shorter structured file id instead of a string 2019-05-17 02:03:23 -07:00
Chris Lu b3b42bc947 replicate need to include new entry path 2019-04-16 00:44:31 -07:00
Chris Lu 97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu 097b7a321e HDFS: move to 1.0.8 2019-02-20 00:57:31 -08:00
Chris Lu 6ed69de6bd HDFS: add jwt auth for uploading 2019-02-19 14:26:59 -08:00
Chris Lu 58d4088db4 HDFS: add tls secured grpc 2019-02-19 11:57:25 -08:00
Chris Lu 74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu 9123d799b8 refactor 2018-12-27 23:40:23 -08:00
Chris Lu 319ab6d98f refactoring 2018-12-27 23:29:51 -08:00
Chris Lu 253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu 9546104961 HCFS 1.05 2018-12-18 20:23:40 -08:00
Chris Lu 2c72447078 HCFS: bug fix for listEntries 
fix https://github.com/chrislusf/seaweedfs/issues/797
 2018-12-18 20:22:51 -08:00
Chris Lu e716ec9908 HCFS 1.04 2018-12-17 20:40:10 -08:00
Chris Lu cab7fc7738 fix directory creation 
fix https://github.com/chrislusf/seaweedfs/issues/794
 2018-12-17 20:37:55 -08:00
Chris Lu d11cfdb356 paginate through large folders 2018-12-17 01:25:02 -08:00
Chris Lu 21315f709d HCFS 1.0.2 2018-12-14 09:16:21 -08:00
Chris Lu 36d1e739e6 1.11 2018-12-08 00:32:34 -08:00
Chris Lu adf12c8825 refactoring 2018-12-07 02:23:45 -08:00
Chris Lu 1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu 6946c51430 avoid range request for gzipped content 2018-12-07 01:23:30 -08:00
Chris Lu 6b39effe7f add a local test 2018-12-07 01:23:02 -08:00
Chris Lu 09383860e9 HCFS recursively delete files 2018-12-04 22:57:23 -08:00
Chris Lu 060ce1d9a2 HCFS: adjust file modification time to milliseconds 2018-12-04 01:11:13 -08:00
Chris Lu 5f8c8caec6 HCFS: chmod 2018-12-04 01:10:25 -08:00
Chris Lu d5197d6a50 HCFS support chown 2018-12-04 00:48:11 -08:00
Chris Lu 091f6297c4 add debug 2018-12-04 00:28:10 -08:00
Chris Lu aea86cbf89 add @Override 2018-12-04 00:28:03 -08:00
Chris Lu ce566f579e add juint 2018-12-03 22:12:31 -08:00
Chris Lu c85ee7c0fd HCFS: read concatenated files 2018-12-03 22:12:20 -08:00
Chris Lu 4119c61df8 HCFS can read files 2018-12-03 20:25:57 -08:00
Chris Lu 5b50182658 put file is working 2018-12-03 01:37:29 -08:00
Chris Lu d3be8e022f HCFS can ls, mkdir 2018-12-02 22:44:49 -08:00
Chris Lu 4263805c78 resolve java jar dependencies 2018-12-02 15:16:49 -08:00
Chris Lu c6a567acab add SeaweedInputStream 2018-11-29 01:04:40 -08:00
Chris Lu 021f5d689b copy the visible chunks logic from Go implementation 2018-11-26 00:32:04 -08:00
Chris Lu b089f1a492 SeaweedWrite close http client 2018-11-26 00:00:55 -08:00
Chris Lu 11eb014311 SeaweedFileSystem add SeaweedOutputStream to write 2018-11-25 23:49:05 -08:00
Chris Lu f970abf14a WIP: adding SeaweedOutputStream for write 2018-11-25 18:01:57 -08:00
Chris Lu 6ecefad692 SeaweedFileSystem add rename 2018-11-25 14:48:17 -08:00
Chris Lu 1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu d9871e92d2 adjust module name 2018-11-24 03:24:40 -08:00
Chris Lu 76cba561cf starting with hadoop compatible 2018-11-24 03:22:25 -08:00
Chris Lu 2697d6a4fb fix deleting bucket 2018-11-08 22:15:21 -08:00
Chris Lu 70008b093e test deletes 2018-11-08 22:08:11 -08:00
Chris Lu a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00